Hydra和W&B配合

正文

Pitfall #1: Using Hydra's Config with wandb.config

Hydra uses omegaconf as the default way to interface with the configuration dictionaries. However, it is very important to keep in mind that OmegaConf's dictionary is not a subclass of primitive dictionaries (unlike tools like Addict). Hence directly passing Hydra's Config to wandb.config leads to unexpected results on the dashboard. It's necessary to convert omegaconf.DictConfig to primitive dict() type, before passing to wandb.config.

Copy@hydra.main(config_path="configs/", config_name="defaults")
def run_experiment(cfg):

    wandb.config = omegaconf.OmegaConf.to_container(

        cfg, resolve=True, throw_on_missing=True

    )

    run = wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project)

    wandb.log({"loss": loss})

    model = Model(**wandb.config.model.configs)

Pitfall #2: Non-Responsiveness During the Start of Training

This is a known pitfall which sometimes occurs in distributed training scenarios (docs) where wandb's multiprocessing interferes with the multiprocessing provided by Hydra's Multirun, specifically in the distributed launchers like Ray's local cluster.

To solve this, try to changing wandb's multiprocessing protocol either by adding an extra settings parameter to wandb.init as:

Copywandb.init(settings=wandb.Settings(start_method="thread"))

or by setting a global environment variable from your shell:

Copy$ export WANDB_START_METHOD=thread